The present invention relates to systems that include mechanisms operable to receive information and to analyze that information on the basis of a learning mode of operation. The present invention adds improvements to the prior inventions of the present inventor as referenced above. While this prior invention provides adequate self-organizing circuit features, improved performance and reduction in costs can be achieved by the additions disclosed herein. The improvements are of two basic types: those that apply to improved circuit design and those that apply to improved "teaching" of the circuit.
Improved circuit design first includes a method to better allow the circuit elements of a self-organizing circuit to learn new patterns quickly, secondly includes a mechanism by which serial or sequential information can be learned, and thirdly includes mechanisms by which the circuits can be simplified by reducing the number of interconnections within the circuit.
Improved teaching of the circuit includes ways by which the self-organizing circuit can be quickly taught new patterns. First by making each input to a subcircuit compete against the many other inputs to that subcircuit, by weighting each input according to simple Boolean functions, and lastly by incorporating a method by which information can be added to the circuit after the circuit has already learned some information. The circuit makes better distinctions between patterns by incorporating modified subcircuits which are change-sensitive and by making the subcircuit competition be sensitive to change. Pattern recognition includes the ability of a circuit to detect a pattern among variables despite the fact that the pattern is not precisely the same pattern as was previously learned. The variables can be considered as any variable or set of variables from which a signal can be formed, in some way functionally related to the variables considered. The types of variables fall into two broad categories: static variables and time-varying variables. For example, when a color-blind person tries to distinguish between letters or numerals of pastel dots, he is given static variables or static information. Time-varying variables for which patterns might be recognized include audio signals, for example a person trying to distinguish between the dash and dot patterns he hears in a Morse code signal.
Clearly living organisms can accomplish this task of pattern recognition. People can recognize static information such as printed material (as the reader of these very words is now doing) and time-varying information such as how to swing a tennis racket so as to make proper contact with a tennis ball. Lower life forms also have this ability: certain ant species can recognize the foliage cover near their nests to orient themselves; certain moths can recognize the high-pitched sounds of a bat to avoid being captured; and even clams can learn primitive patterns of tactile responses which distinguish food from danger. Living organisms use electrochemical signals in the neurons of their brain or ganglion to perform this pattern recognition function.
While very complicated computers have been built which can do enormous numbers of calculations at speeds far exceeding the simple calculations done by house flies and clams, the ability of such computers to perform pattern recognition at the level of these primitive organisms has not been forthcoming. A major difference is that people tell the computers what to do whereas flies and clams tell themselves what to do. The former are essentially preprogrammed to do certain sequences in attempts to recognize patterns in space or in time while the latter self-organize themselves to "learn" to recognize patterns which are important to them. In each case, a certain amount of information is already known: in the computer it is a programming language (software) plus the myriad of interconnections in its circuitry; in the living organism it is its instincts or programmed patterns plus the myriad of interconnections in its neural circuitry.
Circuits made of wires and transistors and other electronic components could do well to have the ability to self-organize or learn as living organisms do. These circuits could lead directly to a machine which recognizes speech or recognizes handwriting among other tasks. A misconception is that people think but computers do not think--computers do only what people tell them to; however, self-organizing circuits of the type herein described mimic the ability of the brain to think or at least to do a major subtask of thinking which is pattern recognition. Hence, the line between a computer thinking and a person thinking becomes a fuzzy one.
It will be noted that in the last few years considerable research has been devoted to associative memories based on an approach by John Hopfield (see, for example, Proc. Natl. Acad. of Sci., Vol. 81, pp. 3088-3092, May 1984). When "taught" patterns, these associative memories have some of the same properties of the prior patents by the present inventor and the present invention. For example, both methods can take arbitrary input patterns of binary information and detect when one of several patterns has been detected. Both methods use a multiplicity of "voter" subcircuits having simple binary outputs determined by combining neighboring outputs, weighting them either positively or negatively. Both methods are insensitive to noise--the input patterns during learning or recognition tasks may be only approximate copies of the exact input patterns and still detect the correct pattern. In a variation of the Hopfield algorithm Geoff Hinton and Terry Sejnowski use random outcomes of the subcircuits to better allow their networks to stabilize on a particular pattern (Cognitive Science, Vol. 9, 1985), much as the present invention uses random outcomes in eliminating the need for training of intermediate subcircuits.
But here the similarity ends. Hopfield, Hinton, Sejnowski and their colleagues all use "network optimization" methods for training their associative memories. Rather than using local outcomes of nearby nodes to adjust the interactions between subcircuits as does the present invention, associative memories optimize the network in total. Errors in associative memories are detected at the input and output subcircuits and interactions between subcircuits are adjusted based on these results rather than on local competition between the subcircuits. In addition, associative memories can only deal with combinational patterns of inputs: those whose inputs are not time-varying. The present invention can accept either combinational or sequential patterns as inputs and can output either combinational or sequential patterns as outputs.
At the circuit level, associative memories require symmetric, bidirectional links between subcircuits--each subcircuit that is connected to another has a similar link from the second to the first in both sign and value. In the present invention, no restrictions are placed on the mutual interactions between subcircuits. In this sense, the subcircuits behave very much like a neuron in the brain's cortex: information is transmitted in one direction down the neuron's axon. In the present invention as in the prior inventions, information flows through a subcircuit in only one direction, not bidirectionally as in an associative memory.